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1.0  Introduction 

This  document  discusses  the  operational  implementation  of  the  Earth  System  Prediction 
Capability  (ESPC)  that  will  provide  global  environmental  information  to  meet  Navy  and  DoD  operations 
and  planning  needs  from  under  the  sea  to  the  upper  atmosphere.  It  will  be  a  fully  coupled  global 
atmosphere/ocean/ice/wave/land  prediction  system  providing  daily  predictions  out  to  10  days  and  weekly 
predictions  out  to  30  days.  The  Initial  Operational  Capability  (IOC)  is  targeted  for  2018  at  the  Navy  DoD 
Supercomputing  Resource  Center  (DSRC). 

At  the  time  of  this  writing,  the  Navy  DSRC  has  two  identical  (one)  IBM  iDataPlex 
supercomputers  on  the  unclassified  (classified)  side  with  1224  (252)  compute  nodes  and  16  cores/nodes 
for  a  total  of  19,584  (4032)  cores  per  machine.  Presently,  Commander  Naval  Meteorology  and 
Oceanography  Command  (CNMOC)  receives  15%  of  the  unclassified  cycles  (-130  Teraflops  (TF), 
-4400  cores)  at  the  Navy  DSRC  for  operational  systems  and  ESPC  will  be  required  to  fit  within  this 
allotment.  They  anticipate  High  Performance  Computing  Modernization  Office  (HPCMO)  funded 
supercomputer  upgrades  such  that  the  capacity  will  triple  in  FY14  (-400  TF,  -13,200  cores)  and  nearly 
tenfold  by  FY16  with  an  estimate  of -1000  TF  (1  Petaflop  (PF),  -33,000  cores). 


2.0  ESPC  component  descriptions 

The  following  sections  describe  the  individual  components  of  ESPC  as  they  exist  at  the  time  of 
this  writing.  Figure  1  is  a  schematic  of  the  existing  uncoupled  system  and  Figure  2  is  the  system  flow. 


Existing  Uncoupled  System 


HYCOM 
(ocean) 
DA:  NCODA 


NAVGEM 
(atmosphere) 
DA:  NAVDAS-AR 


NAAPS 

(aerosol) 

DA:  NAVDAS-AOD 


Figure  1:  Schematic  of  the  uncoupled  system  at  the  time  of  this  writing.  HYCOM  and  CICE  are 
fully  two-way  coupled,  but  none  of  the  other  systems  are.  The  component  acronyms  are  defined 
throughout  the  text  and  DA  =  data  assimilation.  When  the  system  is  fully  coupled,  the  2D  NCODA 
SST  and  ice  analysis  will  no  longer  be  needed  as  those  fields  will  come  from  HYCOM  and  CICE, 
respectively. 


Manuscript  approved  January  21,  2014. 


1 


Existing  System  Flow 
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Figure  2:  Schematic  of  the  existing  non-coupled  system  flow  at  the  time  of  this  writing. 


2.1  NAVGEM  -  the  atmospheric  model 

The  NAVy  Global  Environmental  Model  (NAVGEM)  version  1.1  was  introduced  as  the  U.S. 
Navy’s  operational  global  weather  prediction  system  in  2013  and  replaces  the  Navy  Operational  Global 
Atmospheric  Prediction  System  (NOGAPS)  which  has  been  in  operation  since  1982.  One  of  the  major 
innovations  of  NAVGEM  is  the  introduction  of  a  Semi-Lagrangian/Semi-Implicit  (SL/SI)  dynamical  core 
which  allows  for  higher  model  resolutions  without  the  need  for  smaller  time  steps.  This  initial  NAVGEM 
operational  transition  has  both  higher  horizontal  (T359)  and  vertical  resolutions  than  its  predecessor, 
namely,  50  hybrid-sigma  levels  in  the  vertical  and  a  horizontal  resolution  of  approximately  37  km. 
Further,  NAVGEM  now  includes  cloud  liquid  water,  cloud  ice  water,  and  ozone  as  fully  predicted 
constituents,  and  an  updated  solar  radiation  and  longwave -radiation  parameterizations. 

The  SL  dynamical  core  in  NAVGEM  finds  the  trajectory  of  the  fluid  motion  that  starts  at  the 
previous  time  step  and  ends  up  at  the  NAVGEM  grid  point  location  following  Ritchie  (1991).  Integration 
by  this  method  removes  the  Courant-Friedrichs-Lewy  (CFL)  limitation  in  conventional  fixed  point 
representations  of  the  dynamical  equations;  however,  high-speed  gravity  waves  associated  with  high- 
frequency  fluctuations  in  the  wind  divergence  remain.  This  is  mitigated  by  incorporating  a  SI  method 
into  the  SL  integration,  where  the  terms  responsible  for  the  gravity  waves  are  identified  and  treated  in  an 
implicit  manner,  thereby  slowing  down  the  fastest  gravity  waves.  NAVGEM  contains  advection  of  cloud 
liquid  water  and  cloud  ice  water,  and  a  2-species  micro-physics  cloud  water  parameterization  based  on  the 
work  of  Zhao  and  Carr  (1997). 
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Another  significant  improvement  in  NAVGEM  is  the  addition  of  the  Rapid  Radiative  Transfer 
Model  for  General  Circulation  Models  (RRTMG)  parameterizations  for  solar  and  longwave  radiation, 
developed  by  the  Atmospheric  Environment  Research  Inc.  (Pincus  et  al,  2003).  RRTMG  includes 
significantly  more  radiation  frequency  bands  in  the  solar  and  longwave  spectra  than  the  previous 
NOGAPS  radiation  parameterizations  and  incorporates  additional  molecular  absorbers  and  emitters.  A 
unique  feature  of  the  RRTMG  is  the  use  of  a  Monte-Carlo  technique  to  compute  the  sub-grid  cloud 
variability  and  the  vertical  cloud  overlap. 

An  official  operational  test  (OPTEST)  of  NAVGEM  1.1  versus  NOGAPS  was  conducted  by  Fleet 
Numerical  Meteorology  and  Oceanography  Center  (FNMOC)  for  the  period  of  6  November  2012  -  18 
December  2012  with  a  statistical  evaluation  based  on  FNMOC’s  standard  global  scorecard.  This 
scorecard  evaluates  the  comparative  skill  of  the  models  based  on  anomaly  correlation  (AC),  mean  and 
root  mean  square  errors  of  16  different  fields  and  observation  types,  including  tropical  cyclone  tracks,  IO¬ 
meter  winds  at  buoy  sites,  1000  hPa  and  500  hPa  AC,  and  winds  and  temperatures  at  radiosonde 
locations,  assigning  a  weighted  positive  score  to  the  model  with  statistically-significant  better  forecasts. 
Improvements  in  all  cat  egories  would  yield  a  skill  score  of  +24.  NAVGEM  scored  a  +14,  the  highest 
score  ever  obtained  for  a  global  model  transition  at  FNMOC.  Historically,  global  model  improvements 
resulted  in  a  skill  improvement  of  +2.  NRL  will  continue  to  upgrade  NAVGEM  with  planned  transitions 
to  higher  vertical  and  horizontal  resolutions,  a  more  computationally  efficient  dynamical  core,  further 
improvements  to  the  data  assimilation  system,  more  advanced  physical  parameterizations,  and  the 
assimilation  of  data  from  recently -launched  satellite  sensors. 

Non-coupled  NAVGEM  1.2  is  now  running  operationally  at  Fleet  Numerical  Meteorology  and 
Oceanography  Center  (FNMOC)  and  outputs  spectral  histories  and  variables  on  a  native  Gaussian  grid 
out  to  180-hours  at  3-hourly  intervals.  Further,  a  series  of  IEEE  binary  fields  are  provided  on  a  fixed 
latitude/longitude  grid  and  fixed  pressure  levels  in  the  vertical  at  both  0.5°  and  1.0°  resolutions  containing 
fields  such  temperature,  winds,  moisture  (in  various  units),  geopotential  height,  surface  fluxes  and 
stresses  as  well  as  parameters  such  as  convective  and  stratiform  rainfall.  For  a  single  tau,  this  translates  to 
approximately  135  GBytes  every  6  hours. 

2.1.1  GEFS  -  the  ensemble  forecast  system 

The  FNMOC  Global  Ensemble  Forecast  System  (GEFS)  is  produced  in  four  steps:  1)  take  the 
NAVGEM  high  resolution  analysis  for  the  current  update  cycle  and  truncate  it  to  the  resolution  used  by 
the  ensemble;  2)  take  the  truncated  analysis  plus  80  6-hour  lead  time  forecasts  from  the  previous  update 
cycle  and  make  80  perturbed  versions  of  the  analysis  (i.e.  ensemble  members)  using  the  ensemble 
transform  (ET)  technique;  3)  run  NAVGEM  forecasts  for  each  of  the  80  ensemble  members;  and  4) 
generate  ensemble  products  including  gridded  in  binary  (GRIB)  files  bundled  for  sending  to  the  other 
NWP  centers. 

Eighty  ensemble  members  are  used  in  the  ET  to  ensure  adequate  spread  in  the  variations  of  the 
perturbations.  The  ET  perturbations  are  computed  over  nine  latitude  bands  to  produce  initial  perturbations 
that  resemble  the  geographic  distribution  of  the  NRL  Atmospheric  Variational  Data  Assimilation  System 
-  Accelerated  Representer  (NAVDAS-AR)  analysis  error  estimate.  Since  a  new  high  resolution  analysis 
is  produced  every  six  hours,  new  perturbations  are  made  every  six  hours  so  they  include  the  latest 
observation  and  flow  information. 
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Long  forecasts  to  the  384-hour  lead  time  are  produced  from  20  members  for  the  00Z  and  12Z 
update  cycles.  Short  forecasts  to  the  6-hour  lead  time  are  produced  from  the  remaining  60  members  for 
use  by  the  ET  in  the  next  update  cycle. 

Forecast  output  consists  of  1°  by  1°  spherical  grids  and  is  available  through  Come-And-Get-lt- 
Product-Store  (CAGIPS).  In  particular,  for  the  variables,  levels  and  forecast  times  specified  by  the  North 
American  Ensemble  Forecast  System  (NAEFS),  the  grids  are  converted  into  GRIB  files,  bundled  by 
ensemble  member  and  forecast  time,  and  sent  to  an  ftp  server  that  can  be  accessed  by  the  National 
Weather  Service  and  the  Air  Force  Weather  Agency  (AFWA)  in  multi-model  ensemble  products. 
Graphics  of  ensemble  mean,  spread,  and  probabilities  are  produced  for  display  on  the  NAVY  Enterprise 
Portal-Oceanography  (NEP-Oc).  In  addition,  each  member  of  the  FNMOC  WW3  wave  forecast 
ensemble  is  driven  by  winds  from  a  NAVGEM  ensemble  member.  The  NAVGEM  forecast  ensemble  is 
also  planned  to  be  an  integral  part  of  the  Hybrid  NAVDAS-AR  data  assimilation  system  assisting  by 
providing  dynamic  estimates  of  the  error  in  the  background  forecasts. 

2.1.2  NAVDAS-AR  -  the  assimilation  scheme 

The  NRL  Atmospheric  Variational  Data  Assimilation  System  -  Accelerated  Representer  has  been 
the  operational  assimilation  system  at  FNMOC  since  2009  and  is  a  4-Dimensional  Variational  (4D-Var) 
assimilation  system  capable  of  handling  in  situ  and  numerous  remotely  sensed  meteorological  data  (Chua 
et  al,  2009).  Satellite  observations  account  for  more  than  65%  of  the  total  assimilated  observations  in 
NAVGEM.  NAVDAS-AR  directly  assimilates  radiances  from  microwave  radiometers  and  from 
interferometers  and  spectrometers  in  the  infrared,  and  bending  angle  from  Global  Navigation  Satellite 
Systems  Radio  Occultation  Radio  Occultation  (GNSS-RO)  profiles.  Geostationary  and  Polar  satellites 
also  provide  Atmospheric  Motion  Vector  (AMV)  information  which  is  treated  as  an  in  situ  observation. 
Some  of  the  in  situ  data  types  include  radiosondes,  ships,  stationary  buoys,  ACARS  (Aircraft 
Communications  Addressing  and  Reporting  System),  and  AMDAR  (Aircraft  Meteorological  Data  Relay). 
Altogether  over  20  million  observations  are  processed,  with  an  average  of  2.2  million  assimilated  by 
NAVDAS-AR  to  provide  an  atmospheric  analysis  from  which  a  new  forecast  cycle  is  begun.  The  default 
radiance  bias  correction  method  for  NAVGEM/NAVDAS-AR  is  a  variational  bias  correction  approach, 
which  estimates  the  bias  predictors  simultaneously  with  the  atmospheric  analysis  during  each  data 
assimilation  cycle  (Dee,  2004).  With  variational  bias  correction  the  radiance  bias  corrections  are 
constrained  by  the  other  observation  types,  the  numerical  weather  prediction  model,  and  the  analysis 
procedure  itself.  NAVDAS-AR  contains  a  tangent-linear  and  adjoint  forecast  model,  which  is  at  a 
reduced  resolution  compared  to  the  full  non-linear  NAVGEM;  as  of  2013  this  reduced  resolution  is 
approximately  100km.  The  observations  are  assimilated  over  a  6-hour  time  period  3-hours  before  and 
after  the  desired  analysis  time.  This  6-hour  period  is  further  divided  into  30-minute  windows  where  the 
observations  are  assimilated  and  fit  to  model  trajectories  valid  at  these  times. 

2.1.3  The  NAVGEM  runstream 

NAVGEM/NAVDAS-AR  runs  three  times  per  update  cycle  (every  6  hours)  at  FNMOC.  The 
preliminary  run  (at  +1:30)  generates  boundary  conditions  for  Coupled  Ocean/Atmosphere  Mesoscale 
Prediction  System  (COAMPS)/NAVDAS,  while  the  180-hour  forecasts  are  initialized  using  the  real-time 
analysis  (at  +3:00).  Late  in  the  12-hr  watch  (around  +8:00),  a  post -time  NAVGEM/NAVDAS-AR  run 
generates  the  background  fields  for  the  next  6-hr  update  cycle. 
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The  NAVGEM/NAVDAS-AR  systems  presently  run  on  the  FNMOC  A2  supercomputer.  The 
NAVDAS-AR  system  runs  on  90  processors  in  roughly  15  minutes  while  the  180-hour  NAVGEM 
forecast  at  T359  spectral  resolution  (roughly  37km)  and  50  vertical  levels  (model  top  ~80km),  referred  to 
as  T359L50,  completes  in  50  minutes  (running  on  180  processors). 

All  Emerald  compute  nodes  run  RedHat  Enterprise  Linux  6.x  Operating  system,  with  24G 
RAM/node.  The  nodes  have  two  sockets  which  contain  Intel  Xeon  Westmere  (6  core)  processors,  but 
there  is  a  mix  of  X5670  @  2.93GHz  and  X5660  @  2.80GHz. 

2.2  HYCOM/CICE  -  the  coupled  ocean/ice  models 

Presently,  the  ocean  and  ice  components  are  fully  two-way  coupled.  These  are  the  HYbrid 
Coordinate  Ocean  Model  (HYCOM)  and  the  Community  Ice  CodE  (CICE),  respectively. 

HYCOM  is  a  primitive  equation  ocean  general  circulation  model  capable  of  nowcasting  and 
forecasting  the  3 -dimensional  temperature,  salinity  and  current  structure  of  the  global  ocean.  Its  grid  is 
uniform  cylindrical  from  78.64°S  to  66.0°S  ,  on  a  Mercator  projection  from  66. CPS  to  47°N  and  curvi¬ 
linear  north  of  this  as  it  employs  an  Arctic  dipole  patch  where  the  poles  are  shifted  over  land  to  avoid  a 
singularity  at  the  North  Pole.  It  employs  potential  density  referenced  to  2000  m  and  includes  the  effects  of 
thermobaricity  (Chassignet  et  al.,  2003).  Vertical  coordinates  can  be  isopycnals  (density  tracking),  often 
the  best  coordinate  in  the  deep  stratified  ocean,  levels  of  equal  pressure  (nearly  fixed  depths),  best  used  in 
the  mixed  layer  and  unstratified  ocean  and  sigma-levels  (terrain-following),  often  the  best  choice  in 
shallow  water.  HYCOM  combines  all  three  approaches  by  choosing  the  optimal  distribution  at  every  time 
step.  The  model  makes  a  dynamically  smooth  transition  between  coordinate  types  by  using  the  layered 
continuity  equation.  The  hybrid  coordinate  extends  the  geographic  range  of  applicability  of  traditional 
isopycnic  coordinate  circulation  models  toward  shallow  coastal  seas  and  unstratified  parts  of  the  world 
ocean.  It  maintains  the  significant  advantages  of  an  isopycnal  model  in  stratified  regions  while  allowing 
more  vertical  resolution  near  the  surface  and  in  shallow  coastal  areas,  hence  providing  a  better 
representation  of  the  upper  ocean  physics.  HYCOM  is  configured  with  options  for  a  variety  of  mixed 
layer  submodels  (Halliwell,  2004).  A  more  complete  description  of  HYCOM  physics  can  be  found  in 
Bleck  (2002)  and  an  application  of  global  HYCOM  within  the  Indonesian  Sea  can  be  found  in  Metzger  et 
al.  (2010).  Two  validation  test  reports  (Metzger  et  al.,  2008;  2010b)  describe  the  verification  that  led  to  it 
being  declared  an  operational  system. 

HYCOM  uses  NAVGEM  forcing  that  includes  these  fields:  air  temperature  at  2  m,  specific 
humidity  at  2  m,  net  surface  shortwave  and  longwave  radiation,  total  (large  scale  plus  convective) 
precipitation,  ground/sea  temperature,  zonal  and  meridional  wind  velocities  at  10  m,  mean  sea  level 
pressure  and  dewpoint  temperature  at  2  m.  The  first  six  fields  are  input  directly  into  the  ocean  model  or 
used  in  calculating  components  of  the  heat  and  buoyancy  fluxes  while  the  last  four  are  used  to  compute 
surface  wind  stress  with  temperature  and  humidity  based  stability  dependence. 

HYCOM  outputs  whole  domain,  3 -dimensional  archive  files  on  its  hybrid  vertical  grid.  These  can 
be  a  daily  average,  or  an  instantaneous  snapshot  at  a  user-defined  frequency.  Currently,  the  3-hourly 
native  grid  snapshots  are  further  interpolated  in  space  to  a  uniform  0.08°  latitude/longitude  grid  between 
±80°  latitude  and  in  the  vertical  to  40  pre -defined  Naval  Oceanographic  Office  (NAVOCEANO)  z-levels 
for  the  following  variables:  sea  surface  height  (SSH),  temperature,  salinity,  zonal  and  meridional  velocity 
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components,  and  these  are  output  in  netCDF  format.  For  a  single  tau,  this  translates  to  approximately  2.9 
GBytes. 


The  Los  Alamos -developed  CICE  model  (Hunke  and  Lipscomb,  2008)  is  two-way  coupled  to 
HYCOM  via  the  Earth  System  Modeling  Framework  (ESMF)  (Hill  et  al.,  2004).  The  sea  ice  and  ocean 
models  use  the  same  grid  configuration  and  pass  information  back  and  forth  every  hour.  CICE  includes 
sophisticated  ice  thermodynamics  such  as  multiple  ice  thickness  layers,  multiple  snow  layers  and  the 
capability  to  forecast  multi-categories  of  ice  thickness  according  to  World  Meteorological  Organization 
definitions.  In  addition,  CICE  has  several  interacting  components  including  a  thermodynamic  model  that 
computes  local  growth  rates  of  snow  and  ice  due  to  snowfall;  vertical  conductive,  radiative  and  turbulent 
fluxes;  a  model  of  ice  dynamics  that  predicts  the  velocity  field  of  the  ice  pack  based  on  a  model  of  the 
material  strength  of  the  ice;  a  transport  model  that  describes  advection  of  the  areal  concentration,  ice 
volumes  and  other  state  variables;  and  a  ridging  parameterization  that  transfers  ice  among  thickness 
categories  based  on  energetic  balances  and  rates  of  strains. 

The  input  forcing  is  the  same  as  HYCOM,  with  the  exception  that  CICE  requires  the  downward 
components  of  the  surface  radiative  fluxes  rather  than  the  net  values  currently  used  by  HYCOM.  CICE 
outputs  instantaneous  snapshots  of  41  ice  (ice  concentration,  thickness,  velocity),  ocean  (surface 
temperature,  salinity,  velocity),  and  atmosphere  (temperature,  radiation)  variables.  A  single  tau  is 
approximately  2.7  Gbytes. 

2.2.1  NCODA  -  the  oeean/ice  assimilation  scheme 

The  Navy  Coupled  Ocean  Data  Assimilation  (NCODA)  is  a  fully  three-dimensional,  multivariate 
(3DVar)  data  assimilation  scheme  (Cummings,  2005;  Cummings  and  Smedstad,  2013)  for  the  following 
ocean/ice  variables:  temperature,  salinity,  geopotential,  vector  velocity  components,  and  ice 
concentration;  all  are  analyzed  simultaneously.  Data  are  selected  for  assimilation  based  on  receipt  time 
(the  time  the  observation  is  received  at  the  center)  instead  of  the  observation  time  so,  any  data  received 
since  the  previous  NCODA  analysis  are  used  in  the  next  analysis.  For  each  data  type  the  user  defines  the 
maximum  age  of  data  to  be  used  in  the  analysis.  All  data  will  not  necessarily  be  synoptic  and  so  they  are 
compared  against  a  time  dependent  background  field  using  the  First  Guess  at  Appropriate  Time  (FGAT). 
Hourly  forecast  fields  are  used  in  FGAT  for  assimilation  of  SST  to  maintain  the  diurnal  cycle,  whereas 
daily  averaged  forecast  fields  are  used  in  FGAT  for  profile  data  type,  both  synthetic  and  real.  NCODA 
can  be  run  in  stand-alone  mode  but  here  is  cycled  with  HYCOM  and  CICE  to  provide  updated  initial 
conditions  for  the  next  model  forecast  using  an  incremental  analysis  update  procedure  (Bloom  et  al, 
1996).  Currently  the  NCODA  ocean  analysis  increments  are  inserted  into  HYCOM  over  a  six  hour 
window  whereas  the  NCODA  ice  analysis  is  directly  inserted  into  CICE.  Corrections  to  the  HYCOM  and 
CICE  forecasts  are  based  on  all  observations  that  have  become  available  since  the  last  analysis.  These 
include  surface  observations  from  satellites,  including  altimeter  SSH  anomalies,  sea  surface  temperature 
(SST),  and  sea  ice  concentration,  plus  in  situ  SST  observations  from  ships  and  buoys  as  well  as 
temperature  and  salinity  profile  data  from  XBTs,  CTDs  and  Argo  floats.  See  Table  13.1  in  Cummings  and 
Smedstad  (2013)  for  a  complete  list  of  assimilated  observations  along  with  typical  data  counts.  All 
observations  must  be  quality  controlled  and  this  is  done  via  NCODA_QC  (Quality  Control)  which  is 
operational  at  NAVOCEANO.  By  combining  these  various  observational  data  types  via  data  assimilation 
and  using  the  dynamical  interpolation  skill  of  the  model,  the  3-D  ocean  environment  can  be  more 
accurately  nowcast  and  forecast. 
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2.2.2  The  HYCOM/CICE  runstream 

The  HYCOM/CICE  runstream  starts  with  the  NCODA  analysis  (performed  once  per  day)  at  tau  = 
-12  hours  with  a  ±12  hour  data  window.  After  the  NCODA  analysis,  HYCOM  is  run  forward  with  the 
analysis  incrementally  inserted  into  the  ocean  model  over  the  first  six  hours,  thus  at  tau  =  -6  HYCOM  has 
fully  ingested  all  the  observational  data.  HYCOM  and  CICE  continue  to  run  in  forecast  mode  out  to  180 
hours.  If  for  some  reason  the  NAVGEM  atmospheric  forecast  is  shorter  than  this,  the  last  forecast  time 
point  is  slowly  blended  toward  climatological  forcing  to  complete  the  180  hour  forecast  time  period.  The 
existing  1/12°  system  runs  on  900  IBM  iDataPlex  cores  and  takes  ~1.0  wall  hour  for  a  complete  day,  ~45 
minutes  for  the  NCODA  analysis  and  ~15  minutes  for  HYCOM/CICE.  We  currently  anticipate  the  1/25° 
system  will  run  on  4800  cores  and  take  ~1.5  wall  hours  for  a  complete  day,  -60  minutes  for  the  NCODA 
analysis  and  -30  minutes  for  HYCOM/CICE,  although  these  numbers  are  subject  to  change  because 
efforts  are  underway  to  parallelize  aspects  of  NCODA’s  data  preparation  software. 

2.3  WAVEWATCH  III™  -  the  wave  model 

WAVEWATCH  III™  (WW3)  (Tolman,  2002,  2007)  is  a  third-generation  wave  model  developed 
at  NOAA/NCEP  which  employs  a  third-order  numerical  propagation  scheme  in  order  to  control 
numerical  diffusion  of  swell.  The  wave  growth  and  dissipation  source  terms  allow  more  rapid  wave 
growth  under  the  influence  of  strong  wind  forcing  than  in  previous  wave  models. 

WW3  solves  the  spectral  action  density  balance  equation  for  wavenumber-direction  spectra.  The 
implicit  assumption  of  these  equations  is  that  the  wind  field,  water  depth  and  surface  current  field  vary  on 
time  and  space  scales  that  are  much  larger  than  the  corresponding  scales  of  a  single  wave.  Furthermore, 
the  propagation  scheme  used  by  the  model  is  conditionally  stable,  which  means  that  the  model  becomes 
inefficient  with  resolution  finer  than  0(1  km). 

The  computational  grid  is  typically  on  a  latitude-longitude  (spherical)  mesh,  where  energy  at  each 
grid  point  is  represented  on  discrete  directions  and  frequency  bins.  Curvilinear  grids  have  been 
introduced  in  the  latest  versions  of  the  model  which  include  a  tested  domain  covering  the  Arctic  region. 
This  latter  domain  will  be  included  to  function  together  with  the  traditional  latitude -longitude  meshes  in 
the  multi-grid  system  described  below. 

2.3.1  Multi-grid  model  description 

The  multi-grid  model  which  has  been  validated  (Chawla  et  al.,  2009;  Rogers  et  al.,  2012)  allows 
for  the  two-way  communication  of  energy  across  domain  boundaries.  Typically,  as  it  was  with  older 
versions  of  WW3,  a  host  model  passes  wave  energy  through  the  boundary  to  a  nest  domain  and  whatever 
happens  within  the  nest  domain  does  not  affect  the  host  grid.  This  can  have  the  effect  of  not  allowing  the 
computational  results  with  significant  events  of  a  high  resolution  model — potentially  using  better  winds 
and  better  bathymetry — to  be  shared  with  the  host  and  other  regions. 

An  advantage  to  running  the  multi-grid  version  of  WW3  is  that  the  domain  configuration  is  more 
efficient,  using  computational  resources  more  where  it  is  needed,  i.e.  minimizing  the  redundant  use  of 
computational  resources.  With  older  model  versions,  the  model  computed  for  all  water  points  in  the  host 
domain  regardless  of  whether  these  points  were  already  covered  by  a  nest.  Now,  the  nest  domain  points 
are  mutually  exclusive  from  others  except  where  there  is  overlap  within  the  buffer  zone  around  the 
boundaries.  In  addition,  it  is  possible  run  together  domains  with  different  grid  types  (specifically 
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curvilinear  grids  vs.  regular  grids)  passing  wave  energy  across  the  boundaries  in  both  directions  (Rogers 
and  Campbell,  2009). 

As  the  name  implies,  the  multi -grid  system  runs  multiple  domains  altogether  instead  of  the 
traditional  approach  of  running  individual  domains  and  passing  boundary  condition  information  to  nest 
domains  and  running  those  separately.  Since  everything  is  together,  the  model  set  up  is  less  tedious 
obviating  the  need  to  specify  individual  points  in  the  host  domain  about  the  nest  to  which  information  is 
to  be  shared.  One-way  nesting  is  accommodated  with  the  passing  of  boundary  conditions  of  spectral 
points  for  domains  used  by  WW3  or  other  wave  models  such  as  SWAN  (Similating  WAves  Nearshore) 
(Booij  et  al.,  1999),  which  is  used  for  nearshore  domains. 

2.3.2  Operational  Implementation 

WW3  is  run  operationally  at  FNMOC  (Wittmann,  2002;  Jensen,  2002)  and  is  currently  in 
transition  to  operations  at  NAVOCEANO  (Rogers  and  Dykes,  2012;  Dykes  and  Rogers  2013).  This 
latter  reference  provides  details  on  the  operational  implementation  at  NAVOCEANO.  At  both  centers, 
the  systems  are  fully  automatic.  The  multi-grid  system  is  to  be  implemented  at  NAVOCEANO  but  not  at 
FNMOC  due  to  the  different  purposes  for  running  the  model  at  each  of  these  centers.  FNMOC  wave 
forecasts  are  continually  issued  for  global  and  large  regional  domains,  whereas  NAVOCEANO  provides 
sea  state  dependent  littoral  products  and  services  for  short  term  support. 

As  soon  as  they  are  available,  wind  fields  from  FNMOC  are  processed  to  force  the  wave  model. 
The  availability  of  the  modeled  wind  fields  is  the  primary  factor  that  governs  when  any  wave  model  can 
begin  to  run  in  any  cycle.  The  regional  wave  model  domains  can  use  the  10-meter  zonal  and  meridional 
velocity  components  of  wind  normally  at  three -hour  intervals  from  COAMPS  (Hodur,  1997)  taking 
advantage  of  mesoscale  dynamics.  Otherwise  winds  from  NOGAPS  and  NAVGEM  can  be  used  for  all 
domains.  In  the  multi-grid  case  all  the  wind  fields  from  various  meteorological  models  must  be  available 
or  substituted  with  other  appropriate  fields  before  the  multi-grid  system  can  start. 

On  domains  where  it  applies,  ice  concentrations  from  CICE  provide  inputs  to  the  wave  model. 
Since,  the  ice  field  does  not  change  significantly  from  one  day  to  the  next,  it  is  not  so  critical  to  update  the 
ice  field  daily  in  larger  domains.  Although  not  applied  at  this  time,  ocean  surface  zonal  and  meridional 
components  of  currents  can  also  be  input  into  the  wave  model. 

Restart  files  are  used  to  maintain  continuity  between  cycles.  No  model  run  for  a  cycle  can  start 
without  either  having  a  restart  from  a  previous  run,  or  by  using  a  cold  start  (i.e.  re -initializing  with 
artificial  conditions).  In  the  case  of  a  multi-grid  configuration,  all  the  restarts  in  the  system  are  made  and 
used  in  tandem.  For  any  one  domain  to  be  removed  from  the  system  a  cold  start  must  be  implemented  for 
all  domains  to  continue,  otherwise  a  void  is  left  which  the  system  cannot  handle.  Adding  domains  on  the 
other  hand  can  be  done  on  the  fly,  since  the  energy  of  the  original  space  over  which  the  new  domain  is 
occupying  is  easily  replaced  with  a  cold  start  with  reasonable  values  for  that  domain. 

The  wave  model  is  run  every  12  hours  forecasting  to  as  long  as  180  hours  at  intervals  available  as 
fine  as  hourly.  The  existing  multi-grid  system  at  NAVOCEANO  runs  on  128  IBM  iDataPlex  cores  and  a 
48-hour  forecast  runs  takes  approximately  22  minutes  of  wall  clock  time.  FNMOC’ s  global -only  domain 
at  0.5  degrees  grid  spacing  for  a  96  hour  forecast  running  on  32  processors  on  a  Nehalem  cluster  takes 


about  7  minutes.  We  currently  estimate  that  a  global  domain  at  1/8°  for  a  96-hour  forecast  can  take  up  to 
5  hours. 


All  models  undergo  pre-  and  post-processing  with  regards  to  the  model  run.  This  processing 
involves  preparing  the  input  data  for  the  model  run  and  taking  the  model  output  and  converting  it  into 
other  formats  such  as  netCDF.  For  the  multi-grid  system,  each  individual  domain  can  be  processed 
before  and  after  as  if  they  were  individual  model  runs.  Field  outputs  are  bulk  parameters  calculated  from 
the  energy  spectra  and  include  significant  wave  height  (SWH),  mean  and  peak  wave  direction,  mean  and 
peak  period,  and  wind  waves.  Point  output  of  the  spectral  energy  and  bulk  parameters  are  also  available 
and  in  fact  the  spectra  are  the  source  of  boundary  conditions  for  the  SWAN  models  run  at 
NAVOCEANO. 

2.3.3  WAVEWATCH  III™  -  data  assimilation 

The  WW3  wave  model  assimilation  capability  has  been  integrated  into  the  NCODA  system  and 
has  been  applied  at  FNMOC  (Wittmann  and  Cummings,  2004),  but  not  at  NAVOCEANO.  A  draft 
validation  test  report  has  been  written  but  not  yet  published. 

The  NCODA  assimilation  is  a  3D-Var  technique  and  is  applied  in  a  sequential  incremental  update 
cycle.  All  altimeter  SWH  observations  falling  within  a  6-hr  time  window  are  used  in  the  analysis.  The 
analysis  background,  or  first  guess,  SWH  field  is  generated  from  a  6-hr  wave  model  forecast.  The 
corrections  are  computed  from  the  SWH  observations  and  added  to  the  model  forecast  to  produce  a 
corrected  SWH  analysis  field.  The  full  wave  model  spectrum  is  then  updated  from  the  corrected  SWH 
analysis  field  by  adjusting  the  model  spectrum  at  each  grid  point  with  a  scaling  factor,  such  that  the 
forecast  SWH  matches  the  analyzed  SWH  from  the  altimeter  measurements.  There  is  no  dependence  on 
the  directions  of  energy  in  the  spectra. 

2.4  NAVGEM-LSM  -  the  land/surfaee  model 

Currently  the  NAVGEM-  Land  Surface  Model  (LSM)  is  a  simple  1-D  NRL  developed  column 
model  containing  4  bulk  soil  layers  described  by  their  water  and  ice  content  and  their  temperature.  The 
vegetation  cover  information  is  parameterized  from  the  USGS  data  base  and  contains  prognostic  variables 
of  canopy  temperature  and  canopy  water  content.  Finally  for  snow  covered  ground  a  snow  temperature  is 
also  specified. 

A  second  option  to  the  NRL  developed  LSM  is  the  community  Noah  Land  Surface  Model 
(Mitchell,  2005).  The  Noah  LSM  has  been  incorporated  in  NAVGEM  but  is  not  the  LSM  currently 
operational  in  NAVGEM  at  LNMOC.  Noah  is  a  stand-alone,  1-D  column  model  which  can  be  executed 
in  either  coupled  or  uncoupled  mode.  The  model  applies  finite-difference  spatial  discretization  methods 
and  a  Crank-Nicholson  time -integration  scheme  to  numerically  integrate  the  governing  equations  of  the 
physical  processes  of  the  soil-vegetation-snowpack  medium.  Noah  has  been  used  operationally  in 
National  Center  for  Environmental  Prediction  (NCEP)  models  since  1996,  and  it  continues  to  benefit 
from  a  steady  progression  of  improvements. 

2.4.1  LIS  -  the  land  surface  assimilation  scheme 

Incorporation  of  the  Land  Information  System  (LIS)  and  its  data  assimilation  component  is 
currently  under  development  within  the  NAVGEM/NAVDAS-AR  framework.  The  object  oriented 
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framework  of  LIS  allows  for  a  direct  interaction  with  the  NAVGEM/NAVDAS-AR  systems.  Full 
documentation  of  the  LIS  system  is  available  at  (http://lis.gsfc.nasa.gov). 

LIS  is  a  flexible  land  surface  modeling  and  data  assimilation  framework  developed  with  the  goal 
of  integrating  satellite-  and  ground-based  observational  data  products  and  advanced  land  surface 
modeling  techniques  to  produce  optimal  fields  of  land  surface  states  and  fluxes.  The  LIS  infrastructure 
provides  the  modeling  tools  to  integrate  these  observations  with  the  model  forecasts  to  generate  improved 
estimates  of  land  surface  conditions  such  as  soil  moisture,  evaporation,  snow  pack,  and  runoff,  at  1  km 
and  finer  spatial  resolutions  and  at  one-hour  and  finer  temporal  resolutions. 

The  fine  scale  spatial  modeling  capability  of  LIS  allows  it  take  advantage  of  Earth  Observing 
System  (EOS)-era  observations,  such  as  Moderate  Resolution  Imaging  Spectroradiometer  (MODIS)  leaf 
area  index,  snow  cover,  and  surface  temperature,  at  their  full  native  resolution.  LIS  features  a  high 
performance  and  flexible  design,  provides  infrastructure  for  data  integration  and  assimilation,  and 
operates  on  an  ensemble  of  land  surface  models  for  extension  over  user-specified  regional  or  global 
domains.  LIS  is  designed  using  advanced  software  engineering  principles  to  enable  the  reuse  and 
community  sharing  of  modeling  tools,  data  resources,  and  assimilation  algorithms. 

The  system  is  designed  as  an  object-oriented  framework,  with  abstractions  defined  for 
customization  and  extension  to  different  applications.  These  extensible  interfaces  allow  the  incorporation 
of  new  domains,  land  surface  models  (LSMs),  land  surface  parameters,  meteorological  inputs,  data 
assimilation  and  optimization  algorithms.  The  extensible  nature  of  these  interfaces  and  the  component 
style  specification  of  the  system  allow  rapid  prototyping  and  development  of  new  applications.  These 
features  enable  LIS  to  serve  as  both: 

•  Problem  Solving  Environment  (PSE)  for  hydrologic  research  to  enable  accurate  global  water 
and  energy  cycle  predictions. 

•  Decision  Support  System  (DSS)  to  generate  useful  information  for  application  areas  including 
disaster  management,  water  resources  management,  agricultural  management,  numerical  weather 
prediction,  air  quality  and  military  mobility  assessment. 

2.4.2  The  NAVGEM-LSM  runstream 

The  current  NAVGEM-LSM  does  not  contain  a  data  assimilation  component.  Efforts  to  couple 
the  NAVGEM-LSM  with  LIS  have  begun.  The  first  step  is  to  spin-up  the  LIS  system  with  NAVGEM- 
LSM  initial  conditions  over  a  long  period  typically  1  -year.  Once  completed  the  LIS  system  is  initialized 
by  the  NAVGEM-LSM  and  NAVGEM  atmospheric  state  after  a  NAVDAS-AR  update  cycle.  The  LIS 
data  assimilation  step  creates  an  analysis  which  can  be  used  for  NAVGEM  and  NAVGEM-LSM  forecast 
used  for  the  subsequent  update  cycle.  In  this  scenario  until  the  data  assimilation  components  are  coupled 
the  initial  conditions  provided  by  FNMOC  to  NAVOCEANO  will  contain  NAVGEM-LSM  forecasts. 


2.5  NAAPS  -  the  aerosol  model 

The  Navy  Aerosol  Analysis  and  Prediction  System  (NAAPS)  is  the  U.S.  Navy’s  global 
operational  aerosol,  air  quality  and  visibility  forecast  model  that  generates  operational,  six-day,  forecasts 
of  aerosol  conditions  worldwide.  NRL  has  developed,  tested,  and  transitioned  NAAPS  to  operations  at 
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FNMOC  for  forecasting  the  concentration  of  the  dominant  atmospheric  aerosols  and  the  subsequent 
effects  on  visibility  for  the  entire  globe.  The  six-day  forecasts  of  sulfate,  dust,  smoke  and  salt  aerosol 
particles  are  used  by  a  wide  range  of  DoD  users,  including  weather  forecasters,  mission  planners, 
operators,  and  scientists,  as  well  as  non-DoD  users.  With  this  model,  we  are  now  able  to  predict  the 
concentration  of  the  dominant  visibility  reducing  aerosol  species  up  to  six  days  in  advance  anywhere  on 
the  globe.  NAAPS  is  particularly  useful  for  forecasts  of  dust  storms  downwind  of  the  large  deserts  of  the 
world:  Sea  of  Japan  and  China  Sea,  Mediterranean  Sea,  and  the  tropical  Atlantic  Ocean.  NAAPS  also 
accurately  predicts  the  fate  of  large-scale  smoke  plumes  originating  from  boreal  and  tropical  forests  and 
the  savannah. 

The  Fire  Locating  and  Modeling  of  Burning  Emissions  (FLAMBE)  database  for  biomass  burning 
is  used  operationally  to  describe  the  hourly  smoke  emissions  for  NAAPS.  The  FLAMBE  project  (Reid  et. 
al.,  2004)  successfully  monitors  fire  activity  and  smoke  transport  for  the  globe.  Half-hourly  GOES-8/-10 
Wildfire  Automated  Biomass  Burning  Algorithm  (WF_ABBA)  fire  products  for  the  Western  Hemisphere 
have  been  produced  at  the  University  of  Wisconsin-Madison  and  implemented  in  smoke  source  functions 
for  NAAPS.  FLAMBE  incorporates  MODIS  fire  products  from  the  University  of  Maryland  in  near  real 
time.  This  data  is  incorporated  into  a  smoke  flux  scheme  for  the  Eastern  Hemisphere  thus  allowing 
NAAPS  to  utilize  a  global  smoke  source  function.  FLAMBE  has  been  successfully  transitioned  to 
operations  at  FNMOC. 

The  forecasts  of  aerosol  concentration  are  distributed  via  classified  and  unclassified  networks  for 
use  by  DoD  forecasters,  operators,  planners,  and  aviators.  NRL  has  also  transitioned  an  associated  model 
(Forecast  of  Atmospheric  and  Optical  Radiative  Properties)  that  calculates  the  fundamental  optical 
properties  of  the  different  aerosol  species  at  wavelengths  of  interest  to  DoD  for  Electro-Optic  (EO) 
propagation  calculations.  These  properties  are  used  by  the  Target  Acquisition  Weapons  Software 
(TAWS)  to  calculate  slant-path  visibility.  Previous  to  NAAPS,  the  user  chose  the  aerosol  load  based  on 
local  conditions  and  had  to  input  the  aerosol  information  by  hand.  Now  the  aerosol  conditions  at  any 
point  in  the  world  up  to  six  days  in  the  future  are  automatically  available  for  use  in  TAWS.  In  another 
application,  the  forecasts  are  used  to  screen  satellite  retrievals  of  sea  surface  temperature  (SST)  for  dust 
contamination  by  NAVO,  thus  improving  hurricane  forecasts. 

2.5.1  NAVDAS-AOD  -  the  aerosol  assimilation  scheme 

NAVDAS  for  Aerosol  Optical  Depth  (AOD)  is  comprised  of  data  quality  control  and  analysis 
elements.  It  applies  the  two-dimensional  variational  (2D-Var)  analysis  technique  to  quality  assured 
MODIS  MOD04  aerosol  optical  depth  product  that  has  been  subjected  to  QC,  QA,  bias  correction  and 
cloud  screening.  Four  major  steps  are  included  in  the  NAVDAS-AOD  process:  (1)  Convert  NAAPS 
mass  concentration  to  x  bZ  (3-D  to  2-D  conversion);  (2)  run  NAVDAS  2D-VAR  to  create  a  new  analysis 
field  xaZ  from  x  b/,  and  x  o/.;  (3)  improve  the  NAAPS  mass  concentration  field  using  xaZ  (2-D  to  3-D 
conversion);  and  (4)  use  the  new  mass  concentration  field  as  an  initial  condition  for  the  next  6-hour 
NAAPS  run.  The  system  is  described  in  Hyer  et  al.  (2011),  Zhang  et  al.  (2005),  Zhang  et  al.  (2008)  and 
Zhang  and  Reid  (2006). 

2.5.2  The  NAAPS  runstream 

The  input  satellite  data  for  NAVDAS-AOD  have  a  latency  of  6  hours  and  are  not  available  for 
analysis  at  the  tau  =  0  analysis  time.  Instead,  an  analysis  is  produced  at  tau  =  -6  using  MODIS  MOD04 
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data  for  the  window  of  tau  =  -9  to  tau  =  -3.  The  6-hour  NAAPS  forecast  valid  at  tau  =  -6  is  used  as  the 
first  guess.  This  new  tau  =  -6  analysis  is  used  to  initialize  a  short  6-hour  NAAPS  forecast  to  produce  the 
apparent  tau  =  0  initial  condition  that  is  used  to  initialize  the  long  144  hour  forecast.  MODIS  fire 
detection  data  from  the  previous  24  hours  are  used  to  prescribe  the  smoke  emissions  for  the  144  hour 
NAAPS  forecast.  The  entire  NAAPS  suite  is  completed  in  less  than  90  minutes  using  12  processors. 


3.0  The  future  ESPC  coupled  system 

A  schematic  of  the  future  ESPC  coupled  system  is  shown  in  Figure  3.  Two-way  coupling  will 
exist  between  most  systems,  however  there  are  some  exceptions.  Presently,  NAVGEM-LSM  does  not 
need  SST  from  HYCOM  or  ice  fields  from  CICE,  but  if  in  the  future  it  does  need  these  variables,  they 
would  be  indirectly  passed  between  coupled  NAVGEM  and  NAVGEM-LSM.  Similarly,  aerosol  loading 
in  the  atmosphere  (via  NAAPS)  will  influence  NAVGEM  shortwave  and  longwave  radiation  passed  to 
HYCOM  and  CICE,  but  there  doesn’t  appear  to  be  a  need  for  direct  coupling  of  NAAPS  with  any  of  the 
other  system  components  but  NAVGEM.  At  this  time  there  is  no  active  research  at  the  Naval  Research 
Laboratory  for  implementing  wave  feedback  to  sea  ice,  and  so  there  is  a  one-way  arrow  between  CICE 
and  WW3,  not  a  two-way  arrow. 


Future  ESPC  Coupled  System 


HYCOM 
(ocean) 
DA:  NCODA 


CICE 

(ice) 

DA:  NCODA 


NAVGEM-LSM 
(land-surface) 
DA:  LIS 


2 


NAVGEM 
(atmosphere) 
DA:  NAVDAS-AR 


ST 


NAAPS 
(aerosol) 

DA:  NAVDAS-AOD 


Figure  3:  A  schematic  of  the  future  ESPC  coupled  system  targeted  for  the  Initial  Operational 
Capability  in  2018. 


12 


Table  1:  Projected  horizontal  and  vertical  resolutions  of  the  individual  ESPC  system  components  at 
the  IOC  in  2018. 


Forecast 

Time 

Scale, 

Frequency 

Atmosphere 

NAVGEM 

Ocean 

HYCOM 

Ice 

CICE 

Waves 

WW3 

Land-Surface 

NAVGEM- 

LSM 

Aerosol 

NAAPS 

Deterministic 
short  term 

0-10  days, 
daily 

20  km 

70  levels 
(T639L70) 

1/25° 
(4.5  km) 
41  layers 

1/25° 
(4.5  km) 

1/8° 

(14  km) 

3/16° 

(21  km) 

3/16° 

(21  km) 

Deterministic 
long  term 

0-30  days, 
weekly 

20  km 

70  levels 
(T639L70) 

1/12° 

(9  km) 

41  layers 

1/12° 

(9  km) 

1/4° 

(28  km) 

3/16° 

(21  km) 

3/16° 

(21  km) 

Probabilistic 
long  term 

0-90  days, 
weekly 

37  km 

50  levels 
(T359L50) 

1/12° 

(9  km) 

41  layers 

1/12° 

(9  km) 

1/4° 

(28  km) 

1/3° 

(37  km) 

1/3° 

(37  km) 

The  fully  coupled  system  is  projected  to  produce  a  10  day  short  term  deterministic  forecast  once 
per  day,  a  30  day  long  term  deterministic  forecast  once  per  week,  and  a  90  day  probabilistic  forecast  once 
per  week.  The  projected  horizontal  and  vertical  resolutions  of  the  system  components  are  shown  in  Table 
1.  It  is  difficult  to  accurately  determine  the  future  computing  capacity  at  the  Navy  DSRC  or  FNMOC,  but 
reasonable  estimates  have  been  put  forth  for  Navy  DSRC  in  the  introduction  out  through  FY16. 
Estimating  the  future  computing  capacity  at  FNMOC  for  the  FY18  timeframe  is  not  known  at  this  time. 
However,  the  design  of  the  system  shall  be  that  of  a  modular  system  and  not  infrastructure  specific  to 
allow  for  fluid  implementation  between,  at  minimum.  Navy  DSRC  and  FNMOC.  The  forecast  length  of 
both  the  short  term  and  probabilistic  forecasts  may  have  to  be  adjusted  to  fit  within  the  resources 
available  for  operational  products  at  the  IOC  in  2018.  The  probabilistic  long  term  forecast  may  be  an 
independent  ensemble  run  or  it  may  be  comprised  of  prior  long  term  deterministic  forecasts.  This  is  still 
to  be  determined. 

3.1  Data  streams 

Each  component  system  will  have  its  own  input  and  output  streams.  Both  FNMOC  and 
NAVOCEANO  will  be  responsible  for  maintaining  this  data  flow. 

3.1.1  Input  streams 

3.1. 1.1  Atmospheric  input 

FNMOC  will  collect,  quality  control,  and  deliver  to  the  operational  system  atmospheric  data  in 
the  correct  format  and  in  a  timely  manner,  and  this  includes: 

1 .  In  situ  data 

a.  Radiosondes  and  Pibals 

b.  Dropsondes 

c.  Driftsonde  (Concordiasi) 

d.  Land  and  ship  surface  observations 

e.  Aircraft  observations 

f.  Synthetic  observations 

2.  Satellite  data 
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a.  Surface  Winds 

i.  Scatterometer,  ASCAT  and  ERS-2 

ii.  SSMI/SSMIS 

iii.  WindSat 

b.  Feature  Tracked  Winds 

i.  Geostationary  (6  satellites) 

ii.  Polar  Orbiters  (AVHRR  and  MODIS) 

iii.  Combined  polar/geo  winds  (C1MSS) 

c.  Total  Water  Vapor 

i.  SSMI/SSMIS  TVAP 

ii.  WindSat  TVAP 

d.  GNSS-RO  Bending  Angle 

i.  GRACE-A 

ii.  5  COSMIC  FM1-6(FM3  has  failed) 

iii.  2  GRAS 

iv.  Terra  and  TanDEM  SAR-X 

v.  CORISS 

e.  IR  Sounding  Radiances 

i.  IASI 

ii.  AIRS 

iii.  CrIS 

f.  MW  Sounding  Radiances 

i.  6  AMSU-A  (Channels  4-14) 

ii.  3  SSMIS  (Channels  2-7,9-11,  22-24) 

iii.  3  MHS  (Channels  3-5) 

iv.  1  ATMS  (Channels  1-15,  18-22) 

g.  Ozone  retreivals 

i.  3  SBUV/2 

ii.  1  OMPS-Nadir  Profiler 

iii.  1  OMPS-Total  Column 

iv.  GOME-2 

h.  Aerosol  Optical  Depth 

i.  2  MODIS  (MOD04) 

ii.  4  AVHRR  (ACSPO) 

iii.  VIIRS 

i.  Fire  Biomass 

i.  2  GOES 

ii.  Meteosat 

j .  Soil  Moisture 

i.  AMSR-2 

ii.  SMOS 

k.  Ocean  Altimeter 

i.  Altimeter  Sea  Surface  Height  Anomaly  (SSHA) 

ii.  Altimeter  Significant  Wave  Height  (SWH) 


3.1. 1.2  Oceanographic  input 

NAVOCEANO  will  collect,  quality  control,  and  deliver  to  the  operational  system  oceanographic 
data  in  the  correct  format  and  in  a  timely  manner,  and  this  includes: 

1 .  In  situ  Data 
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a.  Temperature,  salinity,  profile  data  from  XBT,  Argo,  TAO  moorings,  gliders 

b.  Current  observations  from  HF  radar,  drifters 

c.  Optical  data  from  gliders,  AUVs 

d.  Naval  Ice  Center  ice  edge 
2.  Satellite  Data 

a.  Sea  Surface  Temperature  (SST) 

b.  Sea  Surface  Salinity  (SSS) 

c.  Altimeter  Sea  Surface  Height  Anomaly  (SSHA) 

d.  Altimeter  Significant  Wave  Height  (SWH) 

e.  Sea  surface  color  (optical)  data 

f.  Microwave  ice  concentration 

g.  Satellite-based  heat  flux  estimates 

3.1.2  Output  streams 

In  the  coupled  system,  NAVGEM  forecast  output  implies  the  inclusion  of  output  from  NAAPS 
and  the  NAVGEM-LSM.  The  NAVGEM  output  is  required  for  boundary  conditions  of  COAMPS  and 
WW3.  The  HYCOM  output  will  include  output  from  CICE  and  is  needed  as  sea  state  boundary 
conditions  for  NAVGEM.  Current  NAVGEM  output  at  T359L50  is  approximately  0.8  Gb  per  forecast 
tau,  at  T639L70  this  would  grow  to  approximately  3  Gb.  At  1/25°  resolution,  HYCOM/CICE  output  to 
be  passed  to  NAVGEM  will  be  approximately  0.8  Gb  per  forecast  tau.  The  standard  deterministic 
atmospheric  forecast  model  will  run  out  to  10  days  with  output  typically  written  every  3  hours  at  the 
earlier  times  and  less  frequently  either  6  or  12  hours  after  5  days  of  forecast.  Thus  for  the  10-day  (30- 
day)  deterministic  run,  atmosphere  and  ocean/ice  output  will  total  -228  Gb  (-532  Gb).  Along  with  the 
other  forecasting  system  mentioned  above,  the  output  is  also  fed  to  various  applications  such  as  the 
Optimum  Path  Aircraft  Routing  System  (OPARS)  and  the  National  Unified  Operational  Prediction 
Capability  (NUOPC),  various  Target  Acquisition  Weapons  Systems,  and  many  other  critical  applications. 
Finally  the  NAVGEM  ensemble  run  at  T359L50  will  have  0.8  Gb  of  output  per  ensemble  member  per 
tau,  resulting  in  a  volume  for  all  members  forecasts  totaling  about  4  Tb.  The  NAVGEM  ensemble 
outputs  are  required  for  CAGIPS  maintained  at  FNMOC.  Effective  delivery  of  these  model  output  will 
need  to  be  insured  by  the  implemented  system  design. 

3.2  The  cycling  system 

In  order  to  provide  a  comprehensive  understanding  of  the  challenges  with  executing  the  coupled 
system  across  a  distributed  network,  we  will  address  the  issues  related  to  which  tasks  will  run  at  each 
center  as  well  as  some  of  the  issues  with  job  control  and  file  transfer.  The  computer  center  chosen  to  run 
each  task  is  based  on  the  current  projection  of  resources  in  the  FY18  timeframe.  All  the  forecasts  shown 
in  Table  1  will  be  run  on  Navy  DSRC  resources  by  NAVOCEANO;  however,  there  will  be  a  distribution 
of  the  location  of  where  the  various  Data  Assimilation  (DA)  systems  are  run.  The  DA  systems  for 
NAVGEM,  NAVGEM-LSM,  and  NAAPS  will  be  executed  at  FNMOC.  These  DA  systems  create  the 
initial  conditions  for  the  forecasting  system;  consequently,  the  initial  conditions  for  the  NAVGEM  Global 
and  Ensemble  model  will  be  provided  to  NAVOCEANO  by  FNMOC. 

A  cycling  system  has  no  stalling  point,  so  assuming  a  previous  run  of  all  tasks  has  completed,  the 
tasks  are  described  below  in  a  numbered  order  in  which  they  must  execute.  Tasks  with  the  same  number 
can  run  concurrently,  and  the  preceding  letter  indicates  either  Navy  DSRC  (N)  or  FNMOC  (F). 
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Nl.  NCODA  Ocean/Ice  and  Wave  analyses 

NAVOCEANO  will  run  the  NCODA  analysis  system  on  the  Navy  DSRC  for  HYCOM 
and  CICE  as  well  as  for  WW3.  The  analysis  of  the  ocean,  sea-ice  and  waves  are  used  to  initialize 
the  ocean  conditions  for  forecasts  from  the  coupled  ocean-atmospheric  system.  The  NCODA 
analysis  also  requires  24-hour  forecasts  from  the  previous  run  of  the  coupled 
HYCOM/CICE/NAVGEM/NAAPS/NAVGEM-LSM/WW3  for  the  ocean  and  atmospheric 
background,  and  finally  observational  data  of  the  ocean  over  the  same  24-hour  window.  The 
NCODA  Ocean/Ice  and  Wave  analyses  are  run  once  daily. 

N2.  Coupled  HYCOM/CICE/NAVGEM/NAAPS/NAVGEM-LSM/WW3  Forecast 

The  fully  coupled  HYCOM/CICE/NAV GEM/NAAPS/NAV GEM-LSM/WW 3  requires 
ocean  initialization  from  the  NCODA  Ocean/Ice  and  NCODA  Wave  analysis  and  the  atmospheric 
initialization  from  NAVDAS-AR.  The  forecasts  from  the  coupled  ocean-atmospheric  system  will 
be  used  in  the  subsequent  NCODA  for  both  ocean  and  atmospheric  background  when  analyzing 
Ocean/Ice  and  Waves.  In  addition,  distributed  incrementally  to  FNMOC,  the  ocean  and  sea  ice 
forecasts  are  needed  for  the  various  atmospheric  DA  analysis  systems  run  at  FNMOC. 

F3.  NAYDAS-AR  Atmospheric,  NAVDAS-AOD  Aerosol,  and  LIS  Surface  analyses 

The  NAVDAS-AR,  NAVDAS-AOD  and  LIS  assimilation  system  will  require  ocean 
surface  state,  atmospheric  forecasts,  and  a  background  error  estimate.  The  forecasts  are  required 
to  cover  the  assimilation  window  which  is  6-12  hours  (shown  as  6-hours  in  Figure  4).  Output 
from  a  previous  NAVGEM  ensemble  (only  12-hours  of  forecast  are  needed)  will  provide  a 
background  error  estimate  to  NAVDAS-AR.  NAVOCEANO  will  provide  FNMOC  with 
forecasts  of  ocean  surface  state  from  HYCOM  and  CICE  once  daily,  and  a  background  error 
calculated  from  12-hour  forecasts  from  the  NAVGEM  ensemble  four  times  daily.  The 
background  atmospheric  state  will  be  from  a  deterministic  NAVGEM  run  at  FNMOC.  These 
forecasts  and  background  error  estimate  are  used  within  the  NAVDAS-AR,  NAVDAS-AOD  and 
LIS  systems  along  with  the  various  atmospheric  observational  input  data  to  create  initial 
conditions  for  long  deterministic  and  probabilistic  forecasts  of  the  coupled  ocean-atmospheric 
system  described  in  Table  1. 

F4.  Coupled  NAVGEM/NAAPS/NAVGEM-LSM  forecast 

FNMOC  will  run  a  deterministic  NAVGEM  12-hour  forecasts  (along  with  NAVGEM- 
LSM  and  NAAPS)  which  are  required  for  the  subsequent  NAVDAS-AR  assimilation  cycle  four 
times  daily.  If  computational  resources  are  adequate  in  the  FY 1 8  timeframe,  longer  deterministic 
forecasts  and  an  ensemble  system  may  also  be  run  in  addition  at  FNMOC. 

N4.  NAVGEM  ensemble  forecast 

NAVOCEANO  will  run  the  NAVGEM  ensemble  four  times  daily  producing  the 
background  error  estimates  for  the  subsequent  NAVDAS-AR  run  and  long  forecasts  for  systems 
such  as  NUOPC  and  CAGIPS.  Further,  the  ensemble  will  be  used  to  initialize  the  atmospheric 
state  in  the  NCODA  Wave  analysis  for  the  WW3  run  at  FNMOC. 

F5.  NCODA  Wave  analysis 

The  NCODA  Wave  analysis  system  will  use  a  forecast  of  ocean  state  from  the  fully 
coupled  ocean-atmospheric  system,  atmospheric  state  from  the  NAVGEM  ensemble,  and  ocean 
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wave  height  observational  data  to  produce  a  wave  analysis  and  subsequent  forecast  for 
distribution  by  FNMOC  four  times  daily.  This  is  not  to  be  confused  with  the  NCODA  Wave 
analysis  run  at  NAVOCEANO  to  initialize  the  waves  for  the  long  forecast  by  the  coupled  ocean- 
atmospheric  system  which  is  run  once  daily. 


Future  system  flow 


NCODA  Ocean/Ice  analysis 
NCODA  Wave  analysis 

Coupled  HYCOM/CICE/NAVGEM/NAAPS/NAVGEM-LSM/WW3  forecast 
NAVGEM  Ensemble  forecast 
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Figure  4:  A  schematic  of  the  tasks  and  their  distribution  between  Navy  DSRC  (executed  by 
NAVOCEANO)  and  FNMOC. 


3.2.1  Issues 

3.2.1.1  Transfer  of  atmospheric  data  to  NAVOCEANO 

Transfer  of  the  atmospheric  observational  data  from  FNMOC  to  NAVOCEANO  is  not 
recommended.  The  atmospheric  input  data  streams  are  of  a  much  higher  volume  and  are  highly  more 
time  critical  than  the  oceanographic  counterparts  because  the  atmospheric  timescales  are  on  average  much 
shorter.  As  of  2013,  the  atmospheric  assimilation  systems  process  a  continuous  data  stream  of  5  Gb  per 
update  cycle  (6  hour  window)  and  by  2018  we  would  expect  them  to  grow  to  a  minimum  of  40  Gb. 
Further,  by  the  2025  time  frame  these  streams  are  expected  to  exceed  100  Gb  per  assimilation  window. 
FNMOC  has  an  existing  Bulk  File  Transfer  (BFT)  subsystem  which,  together  with  custom  Distributed 
Processing  System  (DPS)  software,  supports  the  automatic  transfer,  through  the  FNMOC  firewall/DMZ, 
of  data  files  into  and  out  of  the  FNMOC  protected  architecture.  To  reiterate,  these  are  continuous  and 
time  critical  data  streams,  and  the  infrastructure  for  ingestion  (BFT  and  DPS),  as  well  as  satellite  dishes 
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and  the  appropriate  agreements  to  receive  and  provide  data  with  the  various  other  data  distributors  exist 
only  at  FNMOC.  Due  to  these  considerations,  it  will  be  most  efficient  to  do  the  atmospheric  assimilation 
at  FNMOC  and  only  move  NAVGEM  history  files  and  initial  conditions  (as  of  2013  ~0.8  Gb)  to  Navy 
DSRC,  where  NAVOCEANO  will  be  responsible  for  running  the  longer  forecasts  of  NAVGEM  and  the 
NAVGEM  ensembles. 

3.2.1.2  Transfer  of  model  data  to  FNMOC 

The  requirement  to  move  large  amounts  of  data  between  FNMOC  and  NAVOCEANO,  in  a 
very  short  period  of  time  and  within  required  Information  Assurance  (IA)  constraints,  is  the  most 
challenging  aspect  of  the  ESPC  project.  The  transfer  of  the  ocean  surface  state  from  the  coupled  ocean- 
atmospheric  system  to  FNMOC  is  not  deemed  to  be  the  largest  volume  as  it  is  only  the  ocean/ice  surface 
and  currently  estimated  at  -0.8  Gb  per  forecast  tau.  The  volume  which  will  strain  resources  is  that  of  the 
NAVGEM  ensemble  which  is  required  for  CAGIPS,  and  the  WW3  run  at  FNMOC.  The  volume  is 
expected  to  be  roughly  80  Gb  per  forecast  tau  for  all  ensemble  members.  A  possible  solution  would  be  a 
Defense  Research  and  Engineering  Network  (DREN)  111  upgrade  to  allow  FNMOC  Service  Delivery 
Point  bandwidth  to  be  increased  to  OC-24  (1.2  Gbs),  doubling  the  potential  throughput.  It  has  been 
suggested  by  the  Navy  DSRC  Director  that  the  best  solution  would  be  a  more  significant  upgrade  to  an 
OC-48  (2.5  Gbs)  circuit  and  this  could  be  accomplished  at  minimal  additional  cost. 

3.2.1.3  Distributed  job  control/scheduling 

Based  on  information  provided  by  the  HPCMO,  maehine-to-machine  communication  between 
a  DSRC  and  any  outside  system  allows  for  only  one  action/command:  secure  copy  (scp).  This  would 
allow  either  FNMOC  or  NAVOCEANO  to  establish  control  of  ESPC  tasks  using  a  file-based  messaging 
paradigm,  using  CRON  and  the  scp  command  to  send  “messages”  between  the  FNMOC  and 
NAVOCEANO.  The  CRON  timer  would  run  at  a  high  refresh  (2  seconds)  continually  looking  for  these 
messages  and  executing.  This  is  a  massive  backwards  step  in  job  scheduling  and  control  which  would 
best  be  served  by  putting  heads  together  and  finding  a  truly  realistic  solution  within  the  IA  constraints. 

3.2.1.4  Future  of  computing  infrastructure 

Estimates  of  the  Navy  DSRC  computing  infrastructure  are  given  in  the  introduction;  however, 
those  of  FNMOC  are  not  conclusive  for  FY 1 8  at  the  writing  of  this  report.  Upgrades  to  the  FNMOC  or 
further  changes  to  the  Navy  DSRC  computing  infrastructure  would  allow  for  rebalancing  of  tasks.  The 
design  of  the  coupled  system  will  continue  to  take  a  modular  approach  with  the  idea  of  a  fluid 
implementation  of  tasks  between  the  centers  which  will  maximize  efficiency  and  keep  data  transfers  to  a 
minimum.  A  larger  computing  capability  at  FNMOC  would  allow  for  reduction  of  the  largest  volume  of 
data  transfer  which  is  the  running  of  the  NAVGEM  atmospheric  ensemble.  If  in  the  future  an  upgrade  to 
the  FNMOC  infrastructure  is  foreseen,  this  would  remove  the  NAVGEM  ensemble  output,  approximately 
4  Tb  every  six-hours,  from  the  data  transfer  and  greatly  alleviate  the  burden  allowing  for  more  stable 
transfer  of  the  initial  conditions  and  the  forecasts  from  the  global  coupled  systems  between  the  Navy 
DSRC  and  FNMOC. 
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4.0  Concluding  remarks 

A  general  outline  of  the  ESPC  coupled  atmosphere/ocean/ice/wave/land  prediction  system  is 
proposed  for  the  Initial  Operational  Capability  targeted  for  2018.  A  description  of  how  it  will  cycle  at 
both  FNMOC  and  NAVOCEANO  is  included,  although  the  specifics  of  how  the  distributed  job  control 
will  function  are  still  to  be  determined  as  the  system  becomes  more  mature.  A  potential  issue  with  regard 
to  the  transfer  of  model  output  between  the  two  centers  has  been  identified  and  must  be  addressed  in  the 
upcoming  years. 
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